Method and apparatus for multi-dimensional content search and video identification

ABSTRACT

A multi-dimensional database and indexes and operations on the multi-dimensional database are described which include video search applications or other similar sequence or structure searches. Traversal indexes utilize highly discriminative information about images and video sequences or about object shapes. Global and local signatures around keypoints are used for compact and robust retrieval and discriminative information content of images or video sequences of interest. For other objects or structures relevant signature of pattern or structure are used for traversal indexes. Traversal indexes are stored in leaf nodes along with distance measures and occurrence of similar images in the database. During a sequence query, correlation scores are calculated for single frame, for frame sequence, and video clips, or for other objects or structures.

The present application is a continuation of U.S. patent applicationSer. No. 16/240,859, filed on Jan. 7, 2019 and issued as U.S. Pat. No.10,977,307; which is a continuation of U.S. patent application Ser. No.15/290,364, filed on Oct. 11, 2016 and issued as U.S. Pat. No.10,210,252; which is a continuation of U.S. patent application Ser. No.15/078,056, filed on Mar. 23, 2016 and issued as U.S. Pat. No.9,489,455; which is a continuation of U.S. patent application Ser. No.13/432,914, filed on Mar. 28, 2012 and issued as U.S. Pat. No.9,323,841; which is a continuation of U.S. patent application Ser. No.12/141,337, filed on Jun. 18, 2008 and issued as U.S. Pat. No.8,171,030; and which claims the benefit of U.S. Provisional PatentApplication No. 60/944,668 entitled “Methods and Apparatus forMulti-dimensional Content Search”, filed on Jun. 18, 2007, which are allincorporated by reference herein in their respective entireties.

CROSS REFERENCE TO RELATED APPLICATION

U.S. patent application Ser. No. 12/141,163 entitled “Method andApparatus for Providing a Scalable Identification of Digital VideoSequences” filed on Jun. 18, 2008, issued as U.S. Pat. No. 8,229,227,and having the same assignee as the present application is a relatedapplication and hereby incorporated by reference.

FIELD OF THE INVENTION

The present invention generally relates to information retrieval systemsincluding systems related to complex objects, multi-dimensional data,rich media, and video.

BACKGROUND OF THE INVENTION

Natural information can best be described by multi-dimensional featurevectors. For example, to identify objects, or video sequences, orbio-molecular structures, or detect actions and behavior, amulti-dimensional search is required on measurements or features of theobject or structure or sequence that is detected. Some of the videoidentification approaches use motion signatures derived from detectedmotion between frames of a video sequence or description of patches,analogous to visual words, in each frame. Motion signatures for a videosequence can be extracted by using statistical data or object tracking.Another popular method uses a bag of words approach to describe anyimage or sequence. Such an approach describes the regions around akeypoint or selected patches in a frame as words and hence theinformation of a frame or video sequence may be indexed on a word byword basis. This approach uses a keypoint detection algorithm to detectpoints of interest and describe a patch around this keypoint. A wellknown implementation is the scale invariant feature transform (SIFT)algorithm which uses scale invariant keypoint detection and signaturevalues for an area around the keypoint. Another recent algorithm fordetecting keypoints or points of interest is the “Speeded Up RobustFeatures” (SURF) algorithm. Selected patches may be tracked andconnected by visual tubes between frames in some implementations. Visualtubes are abstract tubes connecting the same object across multipleframes. Other video search approaches use color histograms to describean image or image sequence. However, such approaches do not includeunique information about each video and are not generally accurate. Theother drawbacks of conventional video search approaches are the size andcomplexity of the individual signatures generally used, and the absenceof an indexing system for these complex signatures. Together thesedrawbacks impact the size of databases and performance of searching forvideo sequences through multi-dimensional databases.

Current retrieval systems are generally based on massiveparallelization. Documents are organized as one dimensional invertedlists. In a large database with 100 billion (B) documents, a onedimensional inverted index may list as many as 1-10B documents. Further,a multi-dimensional query with 10 inputs will require analysis of allthe associated documents listed. This complexity impacts the update timeto update new entries into the database, query performance, andthoroughness of querying. Current systems usually need to limit the sizeof associated documents for practical reasons. As a consequence, all thedocuments in a database are not generally evaluated. To limit the impactof the above issue on accuracy and performance, most current solutionsrely on a technique for dividing the database into smaller sections andthen evaluating a few of these sections resulting in better accuracy andperformance, but such a techniques are impacted by the size of invertedlist documents, and the accuracy is still limited.

SUMMARY OF THE INVENTION

In one or more of its several aspects, the present invention addressesproblems such as those described above. For example, in videoidentification, traversal indexes are derived from a global shapesignature or signatures of selected frames in the video sequence, andfrom the local signatures of prominent objects, or keypoints identifiedin the video sequence, or from signatures derived from detected motionbetween video frames in a sequence. In general, the architectureincludes associated data and differentiating indexes at the leaf node.

One embodiment of the invention addresses a method of organization of amulti-dimensional video or object database using a compact hash or pivotvalue multi-dimensional vector signature as a traversal index. A robusthash is generated as a traversal index from multiple parametersextracted from a region of interest or keypoint in a frame or from aframe of a video sequence. Multiple associated data or signatures arestored at a leaf node.

Another embodiment of the invention addresses a method to post processsignatures and associated data between a video sequence of interest oran object region associated with a query object or a video sequence toincrease accuracy and confidence of a video sequence match. The distancebetween the signatures of the query and original video featuresincluding a region around a keypoint, or an object or a frame iscalculated. Changes in signatures are correlated between a query and adatabase entry for a matching frame, object, or structure to provide afactor in the sequence correlation score. A sequence correlation in timeis provided using differences in frame numbers between pairs of matchingquery and original video signatures.

Another embodiment of the invention addresses a method of generating alikelihood score for a pair of query frames or regions and correlatingbetween matching frames of a query video and an original video. Acorrelation score is generated based on an individual frame similarityscore. A time correlation is generated using relative differences inframe numbers of the original video and the query video. A correlationbetween the original video and the query video is generated by using achange in signatures of each sequence of frames in the query video andin the original video, wherein the original video is an entry in a videodatabase.

Another embodiment of the invention addresses a method to convertdocuments or activity such as online user session information or anynatural event or activity into multi-dimensional vectors. Documents,events, and activity for learning by inference are classified by amulti-dimensional vector. Certain behavior or next state in an activityare expected, wherein the expected next state or the certain behavior isgenerated by a decision tree or a rule based system that takes as aninput one or more identified documents or classifications.

It is understood that other embodiments of the present invention willbecome readily apparent to those skilled in the art from the followingdetailed description, wherein various embodiments of the invention areshown and described by way of illustration. As will be realized, theinvention is capable of other and different embodiments and its severaldetails are capable of modification in various other respects, allwithout departing from the present invention. Accordingly, the drawingsand detailed description are to be regarded as illustrative in natureand not as restrictive.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1A illustrates a hierarchical representation of a multi-dimensionalobject or video database with traversal nodes constructed fromdifferentiating parts of traversal indexes and the leaf nodes storingassociated data and associated indexes;

FIG. 1B illustrates a multi-dimensional content search system inaccordance with the present invention;

FIG. 1C illustrates some examples of regions of interest used togenerate signatures from, and of some signatures to describe global andlocal features of image or video sequence or object in accordance withthe present invention;

FIG. 2A illustrates a method to correlate a query video sequence to thevideo sequences in a database or to correlate subsequent images in aquery object data cube with the object cubes in the database inaccordance with the present invention;

FIG. 2B illustrates an alternate method to correlate a query videosequence to the video sequences in a database or to correlate subsequentimages in a query object data cube with the object cubes in the databasein accordance with the present invention;

FIG. 2C illustrates an alternate method that takes as an input matchingpairs of query and original video segments, and identifies matchingoriginal video segments for specific segments of the query video inaccordance with the present invention;

FIG. 3A illustrates a post processing method to determine, for thelikely video sequence matching candidate, a confidence factor of thecandidate sequence in accordance with the present invention;

FIG. 3B illustrates a post processing method to determine the likelihoodof a match between segments of query video and original video inaccordance with the present invention;

FIG. 4 illustrates a method used to select database signatures toincrease information content such that the signatures optimize thelikelihood of differentiating between the many video sequences stored inaccordance with the present invention;

FIG. 5 illustrates an alternative organization of the database based oncompare pivots centered around clusters of indexes of generatedsignatures in the database in accordance with the present invention;

FIG. 6 shows an alternative method of converting documents or usersessions into multi-dimensional vectors which can be used to efficientlyperform thorough lookup of similar documents or similar user sessions orsimilar events in accordance with the present invention;

FIG. 7 shows a system application wherein an incoming updated or storedimage sequence is processed to generate multi-dimensional vectors whichare further analyzed for information content before adding them to adatabase in accordance with the present invention; and

FIG. 8 shows a system application wherein the incoming query imagesequence is processed to generate multi-dimensional vectors and used toperform a similarity search against a database for identification of avideo clip or an object in accordance with the present invention.

DETAILED DESCRIPTION

The present invention will now be described more fully with reference tothe accompanying drawings, in which several embodiments of the inventionare shown. This invention may, however, be embodied in various forms andshould not be construed as being limited to the embodiments set forthherein. Rather, these embodiments are provided so that this disclosurewill be thorough and complete, and will fully convey the scope of theinvention to those skilled in the art.

It will be appreciated that the present disclosure may be embodied asmethods, systems, or computer program products. Accordingly, the presentinventive concepts disclosed herein may take the form of a hardwareembodiment, a software embodiment or an embodiment combining softwareand hardware aspects. Furthermore, the present inventive conceptsdisclosed herein may take the form of a computer program product on acomputer-usable storage medium having computer-usable program codeembodied in the medium. Any suitable computer readable medium may beutilized including hard disks, CD-ROMs, optical storage devices, flashmemories, or magnetic storage devices.

Computer program code or software programs that are operated upon or forcarrying out operations according to the teachings of the invention maybe written in a high level programming language such as C, C++, JAVA®,Smalltalk, JavaScript®, Visual Basic®, TSQL, Perl, use of .NET™Framework, Visual Studio® or in various other programming languages.Software programs may also be written directly in a native assemblerlanguage for a target processor. A native assembler program usesinstruction mnemonic representations of machine level binaryinstructions. Program code or computer readable medium as used hereinrefers to code whose format is understandable by a processor. Softwareembodiments of the disclosure do not depend upon their implementationwith a particular programming language.

The various illustrative logical blocks, modules, circuits, elements,and/or components described in connection with the embodiments disclosedherein may be implemented or performed with a general purpose processor,a digital signal processor (DSP), an application specific integratedcircuit (ASIC), a field programmable gate array (FPGA) or otherprogrammable logic components, discrete gate or transistor logic,discrete hardware components, or any combination thereof designed toperform the functions described herein. A general-purpose processor maybe a microprocessor, but in the alternative, the processor may be anyconventional processor, controller, microcontroller, or state machine. Aprocessor may also be implemented as a combination of computingcomponents, for example, a combination of a DSP and a microprocessor, aplurality of microprocessors, one or more microprocessors in conjunctionwith a DSP core, or any other such configuration appropriate for adesired application.

The methods described in connection with the embodiments disclosedherein may be embodied directly in hardware, in a software moduleexecuted by a processor, or in a combination of the two. A softwaremodule may reside in RAM memory, flash memory, ROM memory, EPROM memory,EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or anyother form of storage medium known in the art. A computer-readablestorage medium may be coupled to the processor through local connectionssuch that the processor can read information from, and write informationto, the storage medium or through network connections such that theprocessor can download information from or upload information to thestorage medium. In the alternative, the storage medium may be integralto the processor.

FIG. 1A illustrates a hierarchical representation 100 of amulti-dimensional object or video database with traversal nodesconstructed from differentiating parts of traversal indexes and leafnodes storing associated data and associated indexes. In FIG. 1A, atraversal index associated with elements, such as video sequences 104,stored in a multi-dimensional video database 106, and leaf nodes 110.The video database is traversed via nodes 108 to reach the leaf nodes110 storing associated data and associated indexes. The traversalindexes store differentiating information between different videoframes. A typical video database is indexed as a hierarchy and the nodesof interest are traversed from top root node to the leaf nodes.

An alternate method of indexing avoids tree traversal altogether andaccesses the leaf node directly. This indexing is done by generating adirect address or hash for each leaf node.

In a preferred embodiment, the traversal indexes are a direct address ora hash to each leaf node. In this embodiment, all the traversal indexesare equivalent to the hash or direct address of the leaf node.

A hash is computed from various features of a detailed signature of theinformation within a region of interest. One embodiment would divide aregion of interest into sixteen sectors. Energy associated with eachsector are added together providing a total energy for the sector. Thetotal energy of the sector is compared to a threshold value to generatea hash bit value for the sector. With sixteen sectors, a sixteen bithash value is created. Other features associated with each sector mayalso be used to generate additional bits for the hash value. Variouscombinations of hash bits may also be used. A robust hash is defined bya selection of the information and threshold utilized such that thecalculated hash value is more immune to noise and disturbances due todistortion in a query video as compared to the original video.

The traversal index in one embodiment is a shape signature which may begenerated using a histogram of sectored rings around the center point,chosen at a keypoint. The radius of the rings can be selected by equaldivisions of the radius of the entire region, or using log-polar radiusvalues, or variants combining log and equal divisions. In log-polarradius calculations, the radius of each larger ring will increase as amultiple of a factor. A keypoint is selected using a difference ofGaussian (DoG) method or the Hessian-Laplace method which are knownmethods used in image processing. A known implementation is the scaleinvariant feature transform (SIFT) algorithm which uses scale invariantkeypoint detection and signature values for an area around the keypointas discussed by Josef Sivic and Andrew Zisserman, “Video Google: A TextRetrieval Approach to Object Matching in Videos”, Proceedings of theinternational Conference on Computer Vision, October 2003, pg. 1470-1477and by David G. Lowe, “Distinctive Image Features from Scale-InvariantKeypoints”, International Journal of Computer Vision, 60, 2, 2004, pg.91-110 and both incorporated by reference herein in their entirety.Another recent algorithm for detecting keypoints or points of interestis the “Speeded Up Robust Features” (SURF) as presented by Herbert Bay,Tinne Tuytelaars, Luc Van Gool, “SURF: Speeded Up Robust Features”,Proceedings of the ninth European Conference on Computer Vision, May2006 pg 404-417 and incorporated by reference herein in its entirety.Alternatively, other corner or keypoint detection algorithms may beused. In another embodiment, a compact signature may be used based on asingle bit, or multiple bits to represent each histogram bin, where eachbin represents a sector of rings within the region of interest. Thisapproach generates a compact signature that can be evaluated quickly,and is suitable for use with a very large database.

An important step in video identification is selection of frames forextraction of features. This step is primarily used to reduce the sizeof database and optimize the uniqueness of database information. Videoframes can be selected by a few methods known in industry or furtheroptimized versions of the basic methods. The information within asequence of video frames changes based on a rate of change of action oron scene changes. The information of changes from frame to frame can bemeasured by different means. One known method is to use the sum of meanarithmetic differences (MAD) between frames. The sum of differencesbetween frames can be used as a controlling signal. Frames can beselected by using the minima or maxima within a sliding window and theselected frames can then used for feature extraction.

An alternate method of selecting frames may be performed on a compressedvideo stream itself. The rate of information change can be tracked byvarious means including the sum of differences of the change in framesas well as the number of bytes to indicate the changes.

Another alternate method would track objects in a compressed video andselect frames when the object motion is at minima or maxima or at bothminima and maxima.

In another embodiment, a traversal index is generated by combining twosubsequent signatures of selected frames. For example, a combinedsequence signature may be generated representing two subsequentsignatures. Using the content of two frames makes the combined sequencesignatures highly unique and distinguishable from other sequences in thevideo database.

In another embodiment, a shape signature could be based on a combinedset of histograms. Each set of histograms is generated for each keypointor critical point on contours. Here the orientation considered forhistograms would be based on a perpendicular orientation to the maximumgradient or be based on a line of the strongest contour near thekeypoint.

The signature generation process includes:

-   -   a) finding an orientation at each keypoint;    -   b) selecting an orientation axis based on the maximum of any        parameter including sum of first order gradient, or second order        gradient at a keypoint;    -   c) dividing the area around the keypoint into sectors and rings        and using the orientation axis as the reference axis;    -   d) generating histograms around the keypoint; and    -   e) generating signatures for each keypoint.        Alternatively, in another embodiment, the signature generation        process is as follows:    -   a) selecting points to generate signatures based on another        criteria, such as detection of object or bounding boxes of an        object, and using weighted values on the contours within the        selected region; and    -   b) generating a histogram for the each of the identified regions        of interest and generating a shape signature for the entire set        of signatures.        Alternatively, in another embodiment, the signature generation        process is as follows:    -   a) generating signature information in the selected region of        interest by a weighted combination of various features detected        in each region;    -   b) the features detected in each region can include the        following:        -   i. intensity gradient,        -   ii. phase correlation between pixels on the contour,        -   iii. good continuation of contours,        -   iv. texture classification,        -   v. color similarity, and        -   vi. second order gradient;    -   c) obtain the first and second order coefficients to create a        weight for each feature that achieves optimal recall or another        quality of metric such as product of recall and inverse        precision; and    -   alternately, if the features provide better recall or quality of        metric (recall times inverse precision) when they are used        separately rather than combined in a single output, multiple        signatures need to be created for the selected features or set        of combined features.

In another embodiment, a multi-dimensional compact signature isgenerated by the following steps:

a. for a region of interest, a new image is created that sums up thefeatures extracted at each pixel and a calculated pixel value isgenerated;

b. the same region of interest is divided into sectors;

c. the features for each region are evaluated in the following ways:

-   -   i) by a calculated sum of features in a sector,    -   ii) by a calculated sum of energies in the x and y direction,    -   iii) by a calculated gradient of energies in the x and y        directions.

In another embodiment, the signature generation process is as follows:

a) for a region of interest, a weighted sum of each feature at a pixelis used to generate an output pixel; and

b) weights for each feature are computed based on the most optimalsolution. The metric for the most optimal solution is based on theproduct of recall and inverse precision.

In another embodiment the traversal index is generated using thesignatures in above methods.

In another embodiment the traversal index combines bits generated from aset of weighted feature outputted pixel region images.

In another embodiment, each of the leaf nodes 110 in FIG. 1A having theassociated data and indexes could also include texture information orobject location and size information.

In a preferred embodiment, a global shape signature that highlightslocal features around an object is used to generate a signature for aframe. In one embodiment, this method is used to generate multiplesignature candidates. From among these multiple candidates, one or moreof these local objects or area-based shape signatures are selected whena term frequency (TF), as described in further detail below with regardto FIG. 5, for the particular global signature is large indicating thatthe signature is not very unique.

The term frequency (TF) herein represents the number of times a giventerm appears in that document. This count is usually normalized toprevent a bias towards longer documents, which may have a higher termfrequency regardless of the actual importance of that term in thedocument.

The inverse document frequency (IDF) is a measure of the generalimportance of a term which is obtained by dividing the number of alldocuments by the number of documents containing the term, and thentaking the logarithm of that quotient.

In another embodiment, the shape signature is based on an extractedobject using image segmentation and motion correlation methods andexcludes surrounding background information. Motion segmentationgenerally describes the methods that use motion of an object in asequence of images or video to separate the object from the rest of theimage.

FIG. 1B illustrates a multi-dimensional content search system 120 inaccordance with an embodiment of the present invention. Themulti-dimensional content search system 120 may suitably include aprocessor 124, a storage unit 126 for storing a video database 106, atools database 130, and the like. The processor 124 may be closelycoupled with a monitor 132, keyboard 134, and printer 136 over interface140. Alternatively, the monitor 132, keyboard 134, and printer 136 maybe part of a workstation which is loosely coupled to the processor 124over interface 140. Interface 140 may include a connection to theInternet or to a local intra-net, for example. In addition, theprocessor 124 may be a server or a server farm, for example, which iscoupled to the storage unit 126 having access to the video database 106and a tools database 140. The processor 124 may store amulti-dimensional content search program for operation on the processor124. Alternatively, the processor 124 may store the multi-dimensionalcontent search program to be downloaded to a workstation for workstationlocal operation. The multi-dimensional content search program may bestored as electronic media in the storage unit 126 on a high densitydisk drive, an optical disk drive, or the like. The program as acomputer-readable medium may also be downloaded over a communicationnetwork, such as interface 140, from a remote network device, such asanother server or mass storage unit.

FIG. 1C shows several representations 120 of regions of interest and thetypes of signatures generated from processing video frames. The regionsof interest can be circular rings 121, 123, and 125 or rectangles grids(not shown). The distance between the rings can be equidistant or be onlog-polar scale. The signatures 122 that are generated can also beclassified as global and local. Global signatures describe the overallcontent of the image or frame. The local signatures describe localfeatures within the region of interest.

FIG. 2A illustrates a method 200 used to correlate a query videosequence to the video sequences in a video database or to correlatesubsequent images in a query object data cube with the object cubes inthe database. For every selected frame in the query video sequence, avideo database search is performed. The nearest video frames from thedatabase collection for each query video frame are correlated toidentify the highest likelihood of a video sequence match. Next,matching frames from the database for each query are correlated with thesubsequent matching database frames for subsequent queries till a highcorrelation is obtained. One correlation factor relates similar gapsbetween the query frames to the gaps between the database matchingframes. If the likelihood of a sequence match is high, as determined bythe correlation, the likely candidate video sequence is selected forfurther processing.

In a preferred embodiment, a given set of query indexes and signatures201 in FIG. 2A are derived from a query video sequence and used toidentify a similar video sequence. For each query video sequence,certain frames are identified. For each of these selected framessignatures are generated for certain extracted features of the frame orframes around the selected frame. For each of the signatures, atraversal index is also generated. This traversal index is used toaccess the database efficiently. Also, the database of signatures oforiginal videos is also indexed by the traversal indexes computed. Theword traverse is typically used to describe the operations that involvethe stepping or traversal from node to node of the database until theindividual elements of the database are reached. The traversal indexesand the signatures are computed from features such as, the shape,motion, first and second order gradients in the sectors or otherfeatures or combination thereof, to identify likely frames or videosequence in step 202 of FIG. 2A. For each of the signatures andtraversal indexes of the query, a range or a nearest neighborsearch/query is performed. This database search operation involvesdatabase traversal and a list of likely entries in the database areidentified that are within the search criteria. At step 203, asimilarity search computation is performed, which involves reading theleaf nodes for associated data. Then, in step 204, the distance or errorbetween the individual query signatures and database signatures iscomputed.

The distance measure is generally defined as L_(p) normalized where p>1and L_(l) normalized is the sum of differences between a query signaturevector (Q) and an original video signature vector (O) for each dimensionof the signature vector. For example, L_(l) (Query, Original)=sum(Qi−Oi) for all dimensions in a signature vector. Where Qi is the valueof the query vector for the original video feature/frame in a givendimension i, and Oi is the value of the original video feature/framevector for a given dimension i. Another distance measure is aMahalanobis distance which is a statistical measure and takes intoaccount correlations between variables.

Then, the operations of computing a correlation score between anypotential segment of a query video sequence or a frame of the queryvideo with the original video are performed in step 205. This stepinvolves further correlations calculations to extend the correlationthat is initially found when the signature level correlation for thequery and original video is performed in step 204. This correlation isperformed on sequences of query and original video frames or betweenspecific features of the query and original video, or between queryvideo frame and original video frame features. Additional computationscan also be performed by using additional indexes or signatures such astexture, motion, and associated data such as location and size. Theabove correlations will identify a small set of likely matching videosequences or frames. For each likely matching video, the probability ofmatching between query and original video is calculated and acorrelation score is generated in step 206. As described above, thesignature level correlation scores from step 204 identify similar videoframes between a query and an original video. In step 205, a sequence ofsignatures is correlated to increase the probability of a match betweena query video sequence and an original video or a query frame, assumingmany features signatures are available for the query frame, and theoriginal video frame. Step 206 is analogous to a false positiveanalysis. For every likely matching of the original video with the queryvideo, a more detailed correlation between the query video and originalvideo is performed. This false positive analysis is performed betweenthe matching video segments or matching video frames or various videofeatures.

In a preferred embodiment, the computation of the correlation score of asequence based on time correlation is described below.

corr_score_Q0_DB0 is the correlation score between a query video segmentand original video segment.corr_score_Q0_DB0=Σ(max(Eij*((Si−sigma)(Sj−sigma)/K)*(1−DTij)²

where Eij=entropy between correlated queries i and j

-   -   Si=the similarity score of item “i” of the matching sequence        between query and original video signatures    -   Sj=the similarity score of item “j” of the matching sequence        between query and original video signatures    -   Sigma=the threshold score    -   The summation is from the first element to the last of matching        signature pairs in a video sequence. Each signature pair        consists of a query signature and an original video signature        and their associated frame numbers.    -   DTij=is the frame correlation between queries i and j and the        associated original video frames for the queries i and j        DTij=|(QFRj−QFRi)−(DBFRj−DBFRi)|/((QFRj−QFRi)+(DBFRj−DBFRi))        -   where query j>query i; and where j is the next element that            has a valid DB match in the query series: 0, 1, 2, . . . i,            . . . j . . .        -   a valid DB match is defined where (Si-sigma)>0 and the            DTij>0.1        -   and where K is a constant

corr_score_Q0_DB0 is the correlation score between a query video segmentand original video segment.seq_score_Q0_DB0_WIN1=sum(max(Eij*((Si−sigma)(Sj−sigma)/L)*power((1−(DTij),2))+A)

-   -   where L, A are constants    -   and where for WIN1: is a sequence window length; the sequence        length is a programmed value that represents the length of the        matching sequence. The threshold values for given sequence        window length have been found experimentally or through        learning.

Thresholding for sequences defined by a non-linear approximation

-   -   For given sequence window W        Thresh=RATE*power((WIN),NL)        -   where RATE is constant;        -   where NL is constant≈0.5

In a preferred embodiment, the correlation score for a matching videosequence is computed as follows: compute a correlation score between twoclose matches of database frames of the same video sequence. Use theindividual frame similarity score, use the frame distance correlationbetween query and database, and correlate the direction of changeinformation in the query signatures to that of the candidate frames inthe video sequence. Apply an appropriate weight on the query indexes.For unique information content, use the uniqueness of each signature, TFin video database 106 and the distances between the signatures in thequeries.

FIG. 2B illustrates a video search method 220 used to correlate a querysignature sequence, obtained from a query video with the signaturesequence in a video database. For every query video signature 221, avideo database similarity search 222 is performed on databases, such asdatabase 223. The nearest video frames signatures, also referred to as acandidate list in 224, are combined with candidates from searches withother signatures for a given query frame to form a combined candidatelist 225. Some of these pairs in the combined list are selected in step226 to be starting points of potential sequences. In step 226, thematching signature pairs with scores above a certain threshold or thosewhich are in the top “n” list are admitted as the starting point of anew sequence. Next, in step 227, candidates from the combined candidatelist for each query are correlated with the potential sequencesdetermined in step 226. In step 228, a detailed sequence or frameanalysis is performed by combining various sub-segments of correlatedframes or frame segments of the query and the original video. Sequenceswhose score are above a first threshold are combined and evaluated instep 230. Sequences that are greater than a second threshold areevaluated in step 229. In step 229, a false positive analysis isperformed for likely matching sequences obtained from step 228. In step230, a thresholding decision is made on the likelihood of a matchingsequence or sequences for both the combined sequences above the firstthreshold and the sequences above the second threshold that have passedthe false positive test in step 229. Step 231 reports the results andselected results may be displayed in step 232 which shows a sampleresult list having a matching video name, a query start frame (Q St Fr),a query end frame (Q End Fr), an original video start frame (DB St Fr),an original video end frame (DB End Fr), and a likelihood of a match asa confidence value.

FIG. 2C illustrates a video search method 240 used to evaluate a set ofmatching query signature sequences from a query video and matchingoriginal video sequences. For every query video sequence detected instep 241, a detailed sequence or frame analysis is performed in step 242on the sub-segments of correlating frames or frame segments of query andoriginal video. In step 243, the surviving detected sequences arecombined into a list. In step 244 the combined sequences score isevaluated to determine if it is greater than a threshold. If so, the setof sequences of a given video are selected as a matching video. Thescores for each video are evaluated in step 245 to determine the bestmatching video. The best matching video list is generated in step 246.The matching videos are added to a threshold and reporting unit in step248. Step 247 performs false positive analysis on a set of best matchingsequences. Selected results may be displayed in step 249.

FIG. 3A illustrates a post processing method 300 utilized to determine aconfidence factor for a likely video sequence matching candidate of thecandidate sequence. All the signatures of the candidate video, startingwith an identified start frame, for each frame thereafter, and eachidentified frame or sequence, are compared with all the databasesignatures related to the query video sequence and correlation scoresare generated.

One embodiment describes a method to correlate signatures within a videosequence to a database set to identify the likely matching videosequence. The embodiment also describes the method to correlate likelymatching video sequences with all related signatures in database todecide a true match or confidence of the match.

A preferred embodiment of the post processing method 300 to increase theconfidence of a video sequence matching candidate from database is shownin FIG. 3A and described below. A given set of query indexes 301 Aderived from a query video sequence is received in step 301. In step302, a database index is traversed for this set to access leaf nodes forassociated data and associated indexes which are stored in step 303.Then, as described with regard to step 204 of FIG. 2 above, the distancebetween the individual query index and candidate database index andassociated data is computed in step 304. An edit distance can becomputed to obtain a more accurate correlation. For a shape signature,the edit distance is, for example, the sum of the pixel weights and thedistance that needs to be traveled to make two signatures equal. Next, acorrelation score for the single index or frame is computed in step 305using additional indexes such as texture, motion, and associated datasuch as location and size to correlate each individual frame. Thesequence probability scores for each query sequence or frame arecalculated, as well as a correlation score in step 306 for the candidatesequence from various likely start frame positions. Using the abovecorrelations of matching sequences, a final evaluation of thecorrelation between the candidate video and the query video is performedin step 307 to produce a video sequence likelihood. The likelihood isbased on a score that is thresholded by a value that is calculated fromthe length of the sequence or the total information extracted from thevideo sequence. Using rules based on learning a probability, aconfidence value is placed on the likelihood of a match.

FIG. 3B illustrates a post processing method 320 employed to determineif a matching segment of an original video 322 and a query video 321 aresimilar. All the signatures of the candidate video, starting with anidentified start frame, for each frame thereafter, and each identifiedframe or sequence, are compared with all the database signatures relatedto the query video sequence and correlation scores are generated.

The sequence probability scores are calculated for each query sequenceor frame and a correlation score is also calculated in step 323 for thecandidate sequence from various likely start frame positions. Thecorrelation scores calculated are compared in step 324 with a thresholdthat takes into account the total query information, for exampleFn{scores, query_dist}>threshold. If the scores of the video sequenceare greater than the threshold, the sequence is added to a list ofmatching sequences. The results are reported in step 326 and may utilizeother analysis and decisions provided by step 325. Step 327 operates todisplay a rendering of results, where Q St Fr represents a query startframe, Q End Fr represents a query end frame, DB St Fr represents anoriginal video start frame, DB End represents an original video endframe.

FIG. 4 illustrates a method 400 that may suitably be employed to selectdatabase signatures that are more unique so as to increase informationcontent. These selected signatures optimize the likelihood ofdifferentiating between the many video sequences stored. In a preferredembodiment to select high information signatures in the database, videoframes 401 or objects 402, after image processing treatment of the videoframes or objects, are further processed in step 403 to generatesignatures. These signatures are compared in step 406 with databasesignatures accessed in step 410 from a video database, for example.Signatures with high information content relative to the rest of thesignatures in the video database are retained. Based on the uniquenessof the signatures and other control parameters, such as priority of avideo sequence, or total signatures present per video sequence, theselected keys are retained at output 407 and stored in step 410 to thevideo database.

One embodiment describes a method to select database information withhigh uniqueness. If the term frequency (TF) of signatures within a verysmall distance of the generated signature is large, this signature isnot preferred. Another signature that includes more unique informationis preferred for selection, the uniqueness is directly related to thenumber of similar signatures within a given distance measure.

In another preferred embodiment, two pass operations are performed togenerate a high information content database. In the first pass, allcandidate signatures are generated. Next, the total unique informationcontent of each video sequence is evaluated. In the second pass, allsignatures that do not have high information content and at the sametime do not diminish the total unique information about each videosequence are not kept in the database. The database retains primarilyhigh information content signatures that retain most of thedifferentiating information of each database element or video sequence.A measure of the uniqueness of each individual signature and a sum ofunique signatures measured for a particular object or video clip aretracked so as to ensure sufficient information content is in database tobe able to identify the video clip or object. In order to make a measureof unique information content within a query, or more specifically queryvideo, is important to determine the error bounds of a matchingoriginal. For example, if the information content calculated by asummation of uniqueness of individual signatures within a selectedsegment of the query video is very high then the relative error boundsof a matching original video may be relatively high. This form ofmeasurement is based on statistics and can be observed with actual testdata.

One embodiment describes an architecture where the database ispartitioned by pivots. Each pivot is clustered around a centre. Thetraversal indexes are built from each of the pivot. Alternatively eachcluster could be described as a hash or locality sensitive hash value ora cluster centre value. The traversal indexes are built from the clustersignatures or values.

FIG. 5 illustrates an alternative organization 500 of the database basedon compare pivots centered around clusters of indexes of generatedsignatures in the database. In one preferred embodiment of the databasearchitecture, the database is organized around pivots which are used asproxies for centers of each database cluster. The incoming query indexes501 are compared against the compare Pivots 502 and then a treetraversal is followed though nodes 503 to leaf nodes 504. As used hereina pivot in a database refers to a central point around which otherdatabase points are organized.

FIG. 6 shows an alternative method 600 of converting documents or usersessions into multi-dimensional vectors which can be used to efficientlyperform thorough lookup of similar documents or similar user sessions orsimilar events. This method could be used for any input includingdocuments, events, images, natural situations or the like. The firststep in using the input is to classify it into various aspects whereeach aspect provides a dimension. After classification, amulti-dimensional vector is used to describe the values for each class.For example, such values include a document type, such as a shoppinglist, a scholarly paper, a product review, a feedback document. Inanother example, a category may include, for example, a medicalscholarly paper or a video of a medical procedure. Further dimensionscould apply to personalities, topics, activities and the like.

The alternative method 600 includes, classifying an incoming document601 into different aspects in step 602. A first set of dimensions 603such as document type, category, classification, personalities, topics,activities are transferred as input to step 604. In a similar manner tostep 602, an incoming user session information 606 may be classifiedinto various activity types in step 607. A second set of dimensions 608such as a combination of sequence of events, for example, a usersession, and a classification of documents selected and of queries aretransferred as input to step 604. The multi-dimensional vectors 603 and608 are converted into numerical terms in step 604 to generate amulti-dimensional vector 605. The advantages of this method include avery efficient ability to add new documents to update a database, tofind similar documents or duplicates and to perform searches ofdatabases.

One embodiment describes a method to select information from variousfeatures to generate signature(s) for each frame. The method describesmethod to weight the features at corresponding x,y coordinates togenerate a weighted segmented output for a set of selected keypoints orregions.

An alternative method 700 includes, receiving an incoming image sequence701 and preprocessing it into different aspects in step 702. Results ofthe preprocessing in step 702 are further processed in step 707 in whichcorrelation processing is performed to identify information between twoimages and to extract motion based information, including correlatedcontours. In step 702, weighted contours and keypoints 703 are generatedand output for further processing. As used herein, weighted contoursdescribe a weighted sum of various features extracted at a point on thecontour. In step 707, motion segmented objects or correlated objects 708are generated and output. Step 707 includes motion segmentation methodsto create real valued contours of the motion segmented objects. In step704, the multi-dimensional inputs from steps 702 and 707 are used asfollows. Step 704 generates signatures for each region of interest,where a region of interest can be around a selected keypoint or aselected object or bounding boxes of a selected object, or for a frame.Selected signature generation methods, including generating shapesignatures or weighted shape vectors, are used in step 704 to generate amulti-dimensional vector 705 for selected regions of interest. In step709, a database search is performed to find uniqueness of each input andto generate information values 710 which are used to select which valuesare stored in step 711 in the final database. The final database is usedfor search operations for a query video.

An alternative method 800 includes, receiving an incoming image sequence801 and preprocessing that sequence into different aspects in step 802.Results of the preprocessing in step 802 are utilized in step 807 wherecorrelation processing is employed to identify information between twoimages and to extract motion based information, including correlatedcontours. Weighted contours and keypoints 803 are generated in process802 for further processing. Correlated objects 808 are generated inprocess 807. The multi-dimensional inputs are converted using weightedcontours and keypoint information to select area or objects of interestand, after normalization for orientation and diameter, are processedinto numerical terms in step 804 to generate a multi-dimensional vector805. In step 809, a database search is performed to find uniqueness. Thenearest matching results 810 are used to generate a correlation scoresin step 811 which are further processed to generate sequence correlationscores in step 812. The likely matching objects or video clips are againevaluated using all database signatures in step 813. This step 813 isgenerally referred to as false positive analysis. For efficient accessof the database for false positive analysis, the database is indexed byvideo and frame numbers. The nearest results for each incoming imagesequence which constitutes the query are stored as scores.

The various illustrative logical blocks, modules, circuits, elements,and/or components described in connection with the embodiments disclosedherein may be implemented or performed with a general purpose processor,a digital signal processor (DSP), an application specific integratedcircuit (ASIC), a field programmable gate array (FPGA) or otherprogrammable logic components, discrete gate or transistor logic,discrete hardware components, or any combination thereof designed toperform the functions described herein. A general purpose processor maybe a microprocessor, but in the alternative, the processor may be anyconventional processor, controller, microcontroller, or state machine. Aprocessor may also be implemented as a combination of computingcomponents, for example, a combination of a DSP and a microprocessor, aplurality of microprocessors, one or more microprocessors in conjunctionwith a DSP core, or any other such configuration appropriate for adesired application.

The methods described in connection with the embodiments disclosedherein may be embodied directly in hardware, in a software moduleexecuted by a processor, or in a combination of the two. A softwaremodule may reside in RAM memory, flash memory, ROM memory, EPROM memory,EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or anyother form of storage medium known in the art. A storage medium may becoupled to the processor such that the processor can read informationfrom, and write information to, the storage medium. In the alternative,the storage medium may be integral to the processor.

Upon reading this disclosure, those of skill in the art will appreciatestill additional alternative systems and methods for a database queryprocessor in accordance with the disclosed principles of the presentinvention. Thus, while particular embodiments and applications of thepresent invention have been illustrated and described, it is to beunderstood that the invention is not limited to the precise constructionand components disclosed herein and that various modifications, changesand variations which will be apparent to those skilled in the art may bemade in the arrangement, operation and details of the method andapparatus of the present invention disclosed herein without departingfrom the spirit and scope of the invention as defined in the appendedclaims.

The invention claimed is:
 1. A computer-implemented method for storinginformation associated with a video sequence in a reference database,the computer-implemented method comprising: obtaining, by a processor,data associated with the video sequence; and for each frame of a set offrames of the video sequence: determining, by the processor, respectiveglobal features of a global region of interest, determining, by theprocessor, respective local features of a respective keypoint within theglobal region of interest, wherein, for multiple frames of the set offrames of the video sequence, the respective keypoints correspond todifferent respective locations within the global region of interest,generating, by the processor, a respective signature using both theglobal features for the frame and the local features for the frame,determining, by the processor, a respective hash value for the framebased on the signature for the frame, and storing, by the processor, thedata associated with the video sequence in the reference database inassociation with the hash value for the frame.
 2. Thecomputer-implemented method of claim 1, wherein the global region ofinterest comprises multiple sectors, and wherein the global featurescomprise features of respective sectors of the multiple sectors.
 3. Thecomputer-implemented method of claim 2, wherein determining the globalfeatures comprises comparing at least one feature of each of one or moreof the multiple sectors to a threshold value to generate a value for thesector.
 4. The computer-implemented method of claim 2, wherein theglobal region of interest is a rectangular grid.
 5. Thecomputer-implemented method of claim 1, further comprising selecting thekeypoint based on features of the global region of interest.
 6. Thecomputer-implemented method of claim 1, wherein the local featurescomprise features of an area around the keypoint.
 7. Thecomputer-implemented method of claim 6, wherein determining the localfeatures comprises dividing the area around the keypoint into sectors,wherein the local features comprise features of respective sectors. 8.The computer-implemented method of claim 1, further comprising selectingthe frame based on a comparison of the frame with other frames of thevideo sequence.
 9. The computer-implemented method of claim 1, whereinthe associated data comprises a video name and an indication of a framenumber of the frame.
 10. A multi-dimensional content search systemcomprising: one or more processors; and a computer-readable mediumhaving stored therein instructions that are executable by the one ormore computers to cause the multi-dimensional content search system toperform functions comprising: obtaining data associated with a videosequence, and for each frame of a set of frames of the video sequence:determining respective global features of a global region of interest;determining respective local features of a respective keypoint withinthe global region of interest, wherein, for multiple frames of the setof frames of the video sequence, the respective keypoints correspond todifferent respective locations within the global region of interest;generating a respective signature using both the global features for theframe and the local features for the frame; determining a respectivehash value for the frame based on the signature for the frame; andstoring the data associated with the video sequence in a referencedatabase in association with the hash value for the frame.
 11. Themulti-dimensional content search system of claim 10, wherein the globalregion of interest comprises multiple sectors, and wherein the globalfeatures comprise features of respective sectors of the multiplesectors.
 12. The multi-dimensional content search system of claim 11,wherein determining the global features comprises comparing at least onefeature of each of one or more of the multiple sectors to a thresholdvalue to generate a value for the sector.
 13. The multi-dimensionalcontent search system of claim 11, wherein the global region of interestis a rectangular grid.
 14. The multi-dimensional content search systemof claim 10, wherein the functions further comprise selecting thekeypoint based on features of the global region of interest.
 15. Themulti-dimensional content search system of claim 10, wherein the localfeatures comprise features of an area around the keypoint.
 16. Themulti-dimensional content search system of claim 15, wherein determiningthe local features comprises dividing the area around the keypoint intosectors, wherein the local features comprise features of respectivesectors of the sectors.
 17. A non-transitory computer-readable mediumhaving stored therein instructions that are executable by one or moreprocessors to cause a computing device to perform functions comprising:obtaining data associated with a video sequence; and for each frame of aset of frames of the video sequence: determining respective globalfeatures of a global region of interest of the frame, determiningrespective local features of a respective keypoint within the globalregion of interest, wherein, for multiple frames of the set of frames ofthe video sequence, the respective keypoints correspond to differentrespective locations within the global region of interest, generating arespective signature using both the global features for the frame andthe local features for the frame, determining a respective hash valuefor the frame based on the signature for the frame, and storing the dataassociated with the video sequence in a reference database inassociation with the hash value for the frame.
 18. The non-transitorycomputer-readable medium of claim 17, wherein the global region ofinterest comprises multiple sectors, and wherein the global featurescomprise features of respective sectors of the multiple sectors.
 19. Thenon-transitory computer-readable medium of claim 18, wherein the globalregion of interest is a rectangular grid.
 20. The non-transitorycomputer-readable medium of claim 17, wherein the functions furthercomprise selecting the keypoint based on features of the global regionof interest.